Sustainability performance evaluation of municipal selective collection systems applied to a case study

Despite the existence of several waste recycling initiatives in Brazil, the national average of recyclable material recovery rates still remains below 3%. In this context, it is necessary to diagnose the performance of systems to implement improvement actions, considering the particularities of each system. In this study, we describe the application of a method capable of diagnosing the effectiveness of selective collection systems by identifying a general sustainability index, calculated based on 16 performance indicators. The study focused on the city of Volta Redonda and was organized into three stages: identification of the performance evaluation method; data collection and calculation of indicators; and calculation of the sustainability index. The index value found for the city was 0.47, which represents an unfavorable level of sustainability. In this case, Volta Redonda is making insufficient investments in its selective waste collection system. The factors that contributed to this low index are: an outdated solid waste management plan, low collection coverage, low recovery rate of recyclable materials, high rejection rate, high selective collection cost, outdated cost of selective collection service, precarious occupational health and safety conditions of members of waste picker cooperatives, and lack of registration of residences for donation of recyclable materials.


Introduction
Good management of selective waste collection systems is essential to maintain public health in cities, especially with the growing generation of solid wastes (Souza et al., 2021).Many cities have implemented selective trash collection systems with the participation of informally and formally organized waste pickers that have substantially reduced the negative social and environmental impacts produced by the disposal of solid wastes (Ibáñez-Forés et al., 2018;Siman et al., 2020).
In general, the selective collection systems in low-and middle-income countries are not implemented in a planned form (Melaku and Tiruneh, 2020).These systems are put into operation without a sufficiently developed recycling market, without programs to educate the public about the adequate separation and packaging of solid wastes, without evaluating the sustainability of different management models, and without controlling the system's costs (Berg et al., 2018;Expósito and Velasco, 2018;Pérez-Lopez et al., 2018;Campos-Alba et al., 2021).Institutional, financial, political and social questions hamper the development of selective collection systems and the effective inclusion of waste pickers and cooperatives, which are rarely supported by the government and are exploited in the recycling market, and whose problems generally involve financial dependence and difficulties of self-management (Dutra et al., 2018;Siman et al., 2020).
The previous studies of Oliveira et al. (2018), Thoden van Velzen et al. ( 2019) and Gadaleta et al. (2022) about selective collection systems in Europe present discussions on how to improve advanced systems from the standpoint of management and operation.These papers evaluate improvements in systems that are already integrated and fully functioning, involving door-to-door collection models and ecopoints as well as other types of treatment, such as mechanical-biological and pay as you throw (PAYT) systems.The PAYT fee is an economic instrument for management of wastes that applies the "polluter pays" principle, where a fee is charged from the public that varies according to the quantity and type of trash that is discarded (Morlok et al., 2017).
Previous Brazilian studies have evaluated municipal selective collection systems by means of sustainability indicators.Among these are Barros and Silveira (2019), Fratta et al. (2019), Ibáñez-Forés et al. (2019), Silva et al. (2019) andZon et al. (2020).Common problems have been identified by performance assessment, such as a lack of complete and reliable information, the low recovery of recyclable wastes in selective collection systems and low residential coverage, as well as waste picker cooperatives operating under precarious conditions.
In this context, performance evaluation is a tool that can provide the necessary support to make well-informed and responsible decisions (Melnyk et al., 2014;Canonico et al., 2015;Valmorbida et al., 2018).By considering environmental and socioeconomic aspects, this tool promotes improvements in selective collection systems, such as increased service quality, reduced operating costs, better management, and increased recovery rates of recyclable materials (Romano et al., 2019;Llanquileo-Melgarejo et al., 2021;Romano et al., 2022;Schuch et al., 2023).
In Brazil, 65.6 million tons of municipal solid waste were collected in 2021, which corresponds to 0.95 kilograms per person that year (SNIS, 2022).Of the total solid waste collected in 2020, 97.7% were sent to final disposal sites (SNIS, 2022).This disposal can be classified as environmentally sound when the site is a properly managed sanitary landfill, as defined in the National Policy on Solid Wastes (PNRS), meaning that is complies with specific operational standards so as to avoid damages or risks to public health and safety and minimize adverse environmental impacts (Brasil, 2010;Veneu et al., 2014;Olivo et al., 2021).
The largest portion (73.3%) of solid wastes collected was sent to sanitary landfills in 2021.The remaining 26.7% were sent to units considered inadequate, such as controlled landfills and open dumps (SNIS, 2022).The average coverage rate of trash collection for urban and rural areas was 89.9% in 2021.The average indicators of Brazil's five macro-regions varied from 79.0% in the North to 95.8% in the Southeast.It was thus estimated that 22 million people nationwide did not have access to regular door-to-door collection service (direct collection) or collection in containers, bins or dumpsters (indirect collection).In 2021, the number of people without access to collection services increased by nearly 6% in relation to 2020 (SNIS, 2022).
In the case of selective collection, the number of municipalities with some type of formal arrangement was 1,567 in 2021, equal to 28.1% of all municipalities in the country (5,570).According to the National Sanitation Information System (SNIS, 2022), which receives information from 4,900 municipalities in all five regions of the country, this number was 5.8% lower than in 2020.The SNIS' diagnosis (SNIS, 2022) shows that selective collection initiatives are very incipient in Brazil, and the lack of separation of wastes reflects the overload of the final disposal systems and the extraction of natural resources, which in many cases is nearing exhaustion.
The direct consequence of this is that the recovery rates of recyclable materials, despite the establishment of the National Solid Waste Policy (PNRS) in 2010, remain lower than 3% nationwide (SNIS, 2022).In spite of various initiatives to promote the sector and enable greater utilization of recyclable materials, such recovery rates continue to be low, reflecting the weakness of selective trash collection systems.In 2021, it was estimated that 1.12 million tons or solid recyclable wastes were recovered in Brazil, representing 5.3% of the total potential (SNIS, 2022).
In this scenario, improvement of the performance of selective collection systems is still limited by the low technical and financial capacity of municipalities, as well as by insufficient access to technologies and the low technical training of personnel, which makes performance assessment even more urgent (Ibáñez-Forés et al., 2021;Santos and Van Elk, 2021).Therefore, the objective of this study is to report the evaluation of a selective collection system based on the sustainability index proposed by Besen et al. (2017), which considers varied performance indicators, such as system efficiency, institutional aspects, cost, relations with society and the occupational health and safety conditions of waste pickers to identify weaknesses in these systems.

Methodology
Case study: selective collection system in the municipality of Volta Redonda The performance assessment method of Besen et al. ( 2017) was applied in the municipality of Volta Redonda, located in the Southeastern region of Brazil, more specifically in the state of Rio de Janeiro.In geographical terms and from an economic standpoint, Volta Redonda is strategically located among the three largest Brazilian metropolitan regions -those surrounding Rio de Janeiro, São Paulo and Belo Horizonte.The municipality had an estimated population of 273,988 in 2020, and an area of 182.105 km².In terms of population, it is the largest city in the southern region of the state of Rio de Janeiro and the third largest in the state's interior (IBGE, 2020).Figure 1 presents the current selective collection model of Volta Redonda.
There are freelance waste pickers active in all of the municipality's districts.They typically collect the recyclable materials with higher value ("cherry picking").The recyclable materials with lower value are collected regularly and transported to a sanitary landfill located in the neighboring municipality of Barra Mansa, which also receives materials from districts in that city not yet served by selective collection, or they are sent to the facilities of three cooperatives established in Volta Redonda.
In the areas they serve, these cooperatives are responsible for the entire selective collection operation, involving transport, storage, sorting and sale of the recyclable wastes.The door-to-door collection model is adopted.The three cooperatives work in warehouse buildings supplied by the municipal government, which also pays the monthly water and electricity bills.The relationship between the municipal government and the cooperatives is regulated by a contract with each one.The cooperatives' revenues are comprised of the fee paid by the municipal government for selective collection (service provision) and the amount received from the sale of the recyclable materials to intermediaries.

Steps of the sustainability performance assessment
Sustainability performance was evaluated based on the method developed by Besen et al. (2017) to determine the sustainability index, as presented in Figure 2. In Step (1), we conducted a literature review of articles at the Coordination for the Improvement of Higher Education Personnel (Capes) Periodicals portal, which is linked to other research databases, such as Web of Science (WoS) and Scopus.In Step (2), we calculated a range of indicators from primary and secondary data, by applying the following methods: documental research, in loco observation, and interviews.In Step (3), we used the value of each indicator to calculate the sustainability index.
Step 1: literature review At the search page of the Capes Periodical portal, we typed in the keywords "indicators" and "waste management" to find articles with these words in the English title (regardless of the language of the article), published in the previous five years (2017 to 2021).This search resulted in 82 articles.We checked the content of the   Therefore, to be more complete, we utilized the method described by Besen et al. ( 2017), which includes 16 sustainability indicators, presented in Table 1.
To carry out the performance assessment, we used the parameters proposed by Besen et al. (2017) for comparison with each indicator.Table 2 presents this set of parameters and also four ranges of values that indicate the four levels of the sustainability index: very favorable, favorable, unfavorable and very unfavorable.
Step 2: examination of primary and secondary data In Step (2), we examined the primary and secondary data to calculate each indicator based on the following methods: documental research; in situ observation; and interviews.The interviews were conducted with support of semi-structured questionnaires about the indicators, with flexibility for the respondents to add comments when they deemed pertinent.Table 3 specifies the research methods used to calculate each indicator.The visits and interviews occurred in January and February 2022.
The value of each indicator was obtained by three methods, as described by Besen et al. (2017).In particular, three rates were obtained from the data made available by the SNIS (2021): selective collection cov-erage rate (indicator ISSC3); recovery rate of recyclable materials (indicator ISSC10); and rejection rate (indicator ISSC11).The SNIS requests and receives data from municipal governments in all regions of the country and makes these data available for consultation and diagnosis.
Step 3: calculation of the sustainability index In Step (3), based on the value of each indicator, we obtained the sustainability index by calculating the weighted average of the 16 indicators applied (Table 1).The weights were assigned to each indicator as suggested by Besen et al. (2017).The index allowed identifying the level of sustainability of the selective collection system according to four levels: • very favorable: meaning that the selective collection system in the municipality is near sustainability or has already attained it; • favorable: the municipal government is making reasonable investments in the sustainability of the selective collection system, but has not yet achieved this goal; • unfavorable: the municipal government is making only a small investments in the sustainability of the selective collection system; • very unfavorable: the municipal government is not making any investments in the sustainability of a selective collection system (Besen et al., 2017).

Results and Discussion
The sustainability index obtained for Volta Redonda was 0.47, a result of the weighted average of the 16 indicators used in this study.The value of the index denoted an "unfavorable" performance, but was near the threshold of 0.50 of a "favorable" sustainability rating.
Table 4 reports the results with the calculation of each sustainability indicator as applied to Volta Redonda.The municipality does not have an approved integrated solid waste management plan in force.However, there are legal instruments (service provision agreements) with three waste picker cooperatives, namely Contract 065/2021 with Reciclar VR, Contract 066/2021 with Folha Verde; and Contract 067/2021 with Cidade do Aço.These contracts were established in 2017 and have been updated since then.
According to the SNIS ( 2021), the regular collection coverage in 2020 was 43.3%.It also showed that all the money allocated to finance waste management was included in the annual budget.According to the interview with the representative of the Municipal Infrastructure Secretariat, 4.5% of the budget goes to waste management.The municipal government has not yet established a fee or other charge for the regular and selective collection services.Resolution 079 issued in 2021 by the National Water and Basic Sanitation Agency (ANA) specifies that municipalities must create a fee or other charge to cover regular trash collection service.In Volta Redonda, the creation of this fee is under study.For selective collection, no date has been set for the creation of any fee or charge.
According to the interview with the representative of the Municipal Environmental Secretariat, of the 11 desirable aspects for education and disclosure of the selective collection program, nine were in practice by the municipal government.Some of these are the conduction of campaigns with the preparation and distribution of flyers and activities with students, municipal employees and community members in general.We also learned of the existence of an "Intersectoral Committee for Monitoring the Selective Collection System", with the participation of the three waste picker cooperatives.This committee was created in 2018 by means of Municipal Decree 14,993 (PVR, 2018).
With regard to the diversity of partnerships articulated, it was possible to verify that, of the ten desirable partnerships, seven are established.Of these seven, five derive from Municipal Decree 14,993 (which created the Intersectoral Committee).An indirect partnership arrangement also exists with the National Association of Waste Pickers of Recyclable Materials (ANCAT), which is the technical arm of the National Movement of Waste Pickers of Recyclable Materials (MNCR).This association has provided assistance to two of the cooperatives, Reciclar VR and Folha Verde.
Through interviews with representatives of the three cooperatives, it was possible to verify that 47 waste pickers are registered.In the city, there are 33 Social Assistance Reference Centers (CRAS), where citizens can apply to receive some type of aid.Of these applicants, 47 declared they were waste pickers.Regarding the number of dwellings that have adhered to selective collection, the three cooperatives did not have this information.
For this reason, we attributed a value of zero to indicator ISSC9.We also noted the low recovery rate of recyclable materials, of only 1.78% in 2020, according to the SNIS (2021).
Further, according to the SNIS (2021), there was a high rejection rate (17.4%) in the municipality in 2020.We also investigated the working conditions during selective collection.Of the 12 desirable aspects, ten were satisfied by the cooperatives.The cooperatives use both rented and their own trash collection trucks.The waste pickers use adequate personal protection equipment (PPE) and wear suitable uniforms.Besides this, the Vehicle Ownership Tax (IPVA) was current and the other documents were valid during the research period.
Another checklist was applied regarding the working conditions at the sorting centers.Of the 13 desirable aspects, only five were satisfied.This checklist revealed the precarious working conditions of the facilities of the cooperatives granted by the municipal government.In the case of Folha Verde, the restrooms were not linked to the sewer system.In the case of Reciclar VR, the men's bathroom had no water.The walls of the buildings had cracks and generally required repair and painting.
The final checklist covered questions about the health and safety of the members.Of the seven desirable aspects, only one was satisfied (use of PPE by the waste pickers).The other basic questions were being neglected, such as the existence of adequate fire extinguishers; an emergency plan; and a workplace accident prevention committee.
With respect to the cost of the selective collection service, the municipal government pays a subsidy to the cooperatives of R$ 625 per ton sold (demonstrated by the sale invoices of recyclable material submitted by the cooperatives), as established in the contracts.Thus, the cooperatives have two sources of revenue: the municipal government and intermediaries (amount paid upon sale of the recyclable materials).The value of R$ 625 per ton was determined by the municipal government after market research and consultation of contracts established by other municipal governments.
Finally, we calculated the ratio between the cost of selective collection and that of regular collection including the cost of disposal, expressed as a percentage.The cost of regular trash collection was R$ 186.58 per ton, while the cost of disposal in the sanitary landfill in the neighboring municipality of Barra Mansa was R$ 69.71 per ton.Therefore, the cost of regular collection plus disposal was R$ 256.29 per ton.The total cost of selective collection was R$ 625 per ton.Thus, the ratio was 243.9%.
We found that more than half of the sustainability indicators were "very unfavorable".After analyzing these indicators, it was possible to obtain a diagnosis of the city's selective collection system.In general, the system is expensive (reflected by the indicators ISSC15 and ISSC16).This cost is twice the average cost of Brazil-ian municipalities (R$ 313.17 per ton) estimated by the association called "Compromisso Empresarial para Reciclagem" (Business Commitment to Recycling) (CEMPRE, 2020).Furthermore, the cost of selective collection is more than twice the cost of regular collection when also considering the cost of final disposal.
In this scenario, the municipal government pays the total cost of selective collection (reflected by the indicator ISSC4).Despite this, the system is unable to achieve good results, including due to low coverage (reflected by the indicator ISSC3).And this low coverage aggravates the low rate of recovery of recyclable materials (reflected by the indicator ISSC10).Moreover, another problem is the high rejection rate (reflected by the indicator ISSC11).Curiously, the actions among the population for education and disclosure of the selective collection service (indicator ISSC5) are "very favorable".This indicates that sufficient actions are being implemented by the municipal government.However, it seems that these actions are not producing the desired effect of encouraging the people to separate their trash for selective collection, which would reduce the rejection rate.This failure to produce an effect applies to the rate of recovery of recyclable materials.From another standpoint, the number of partnerships (indicator ISSC7) is considered "favorable".Nevertheless, the quality of these partnerships is low, hampering the attainment of positive results.
The municipal government has a committee (indicator ISSC6) formed by representatives of the Municipal Secretariats of the Environment and of Infrastructure, along with the Public Defender's Office and Labor Attorney's Office (the last two federal) and the cooperatives.The members of this committee meet once a month.Thus, it is necessary for the municipal government to pay heed to the recommendations of this committee for actions to improve the system.For this reason, it is necessary to have an integrated solid waste management plan (indicator ISSC1) in force, indicating the performance of the system and the results that must be reached.This involves reflection on the responsibilities for waste management.Our results show that these responsibilities for the regular and selective collection service are allocated to different municipal secretariats, which hampers the obtainment of an effective, integrated, solid waste collection.
Furthermore, despite a "favorable" number of cooperatives in relation to enrolled waste pickers (indicator ISSC8) and the fact the cooperatives have formal contracts (indicator ISSC2), the waste pickers who are members of the cooperatives still suffer from the same insecurities and difficulties as the freelance waste pickers.In particular, although the working conditions for collection of recyclable materials (indicator ISSC12) is considered "favorable", the facilities of the sorting centers are in precarious conditions (indicator ISSC13), offering risks to the workers' health and safety (indicator ISSC14).These conditions are not being audited by the municipal government, and the respective contracts lack clauses in this respect regarding collective collection.Indeed, a fire occurred once at the sorting center where the cooperative Reciclar VR now operates.
In previous studies carried out in Brazil's Southeastern region, such as Barros and Silveira (2019), it was possible to identify a municipality with a very unfavorable self-financing indicator and another with a very unfavorable rating regarding an integrated solid waste management plan.It was also possible to identify, in the study by Zon et al. (2020), some factors responsible for the poor performance of the systems: precarious occupational health and safety conditions of waste pickers organized in cooperatives; low recovery rate of recyclable materials; high cost of selective collection; and lack of a fee to finance the collection program.These are problems faced in common by municipalities in different states and in Volta Redonda.

Conclusions
The low sustainability index obtained indicated that the Volta Redonda municipal government is making only small investments in its selective collection system.The factors underpinning the unfavorable level were lack of an integrated solid waste management plan in force; low collection coverage; low recovery of recyclable materials; high rejection rate; high cost of selective collection; no fee indicated for the selective collection service; lack of enrollment and monitoring of residences for donation of recyclable materials; and precarious health and safety conditions of the cooperative members.
To attain the targets established in the National Solid Waste Plan, it is necessary to revise the actions adopted and insert new strategies for the purpose of eliminating the causes of these problems.It is also necessary to establish better integration of the system, involving other municipalities and cooperatives, besides searching for new partners, such as other entities representing waste pickers, schools and community associations.We found that the existence of a contract does not necessarily make the system more integrated.In this respect, municipalities can avoid executing actions by allocating them to the waste picker cooperatives by means of a service provision agreement.
The problem in this regard is that these cooperatives can still be in the structuring phase even after the process of formalization, and cannot execute all the actions required with the proper quality.For this reason, it is necessary for the municipalities to establish contracts that take into consideration the structuring phase of the cooperatives, without neglecting their ability to work in integration, by implementing strategies to strengthen the operation and management of these cooperatives.Since this is an ongoing improvement process, it is important to evaluate the performance periodically as a tool to attain targets.To overcome the limitations detected in this study, referring to the reliability of the information obtained from the interviews, we consulted other sources of information, such as contracts and decrees, for comparison with the information from the interviews with the representatives of the cooperatives and municipal government.Finally, for new studies, we suggest applying the method of Besen et al. (2017) in other municipalities in the state of Rio de Janeiro.With this, it will be possible to compare the performance of different systems, to detect strengths and weaknesses and opportunities for improvement that can be shared with other municipalities and waste picker cooperatives.

Figure 1 -
Figure 1 -Current selective collection model in Volta Redonda, state of Rio de Janeiro.G: generators of recyclable wastes; D: districts where the generators are located; F: freelance waste pickers; SU: sorting units where cooperatives operate; R: recycling intermediaries; I: recycling industries; N: number of generators, waste pickers, intermediaries or industrial companies active in the system; X: number of districts in the city served by selective collection.Figure 2 -Steps for the evaluation of sustainability performance.

Figure 2 -
Figure 1 -Current selective collection model in Volta Redonda, state of Rio de Janeiro.G: generators of recyclable wastes; D: districts where the generators are located; F: freelance waste pickers; SU: sorting units where cooperatives operate; R: recycling intermediaries; I: recycling industries; N: number of generators, waste pickers, intermediaries or industrial companies active in the system; X: number of districts in the city served by selective collection.Figure 2 -Steps for the evaluation of sustainability performance.

Table 2 -Parameters to evaluate each performance indicator.
Besen et al. (2017) sustainability of selective collection; C1.1: Existence of a sanitation or solid waste plan at the intermunicipal/regional/micro-regional level, including selective collection and participative construction, in execution; C1.2: Existence of a municipal plan with participative construction undergoing implementation; C1.3: Existence of a municipal plan without participative construction, not implemented; C1.4:No plan; C2.1: Existence of a service provision agreement with the municipal government; C2.2: Existence of a working arrangement with transfer of financing; C2.3: Existence of a working arrangement without transfer of financing; C2.4:No agreement or working arrangement; C4.1: Charging of a fee to cover the cost of the solid waste collection service, including selective collection; C4.2: Inclusion of a surcharge in the urban property tax[IPTU]or budget to cover the entire cost to the service; C4.3: Inclusion of a surcharge in the IPTU or budget that does not cover the entire cost to the service; C4.4: Fee revenue only in the budget; C6.1: Existence of one or more instances in functioning with participation of waste pickers; C6.2: Existence and functioning without participation of waste pickers; C6.3: Existence of an instance, but not functioning; C6.4:No existence of an instance with participation.Source: parameters defined according toBesen et al. (2017).

Table 3 -Research methods used to attribute a value to each performance indicator.
ISSC: Indicators of sustainability of selective collection; A: Interview with the person in charge of selective collection of the Municipal Environmental Secretariat; B: Interview with the person in charge of regular trash collection of the Municipal Infrastructure Secretariat; C: Interviews with the people in charge of social actions for the Municipal Community Action Secretariat; D: Interviews with the representatives of the three cooperatives (Reciclar VR, Folha Verde and Cidade do Aço); E: Documental research -contracts with the cooperatives, regular trash collection company, including final disposal, Municipal Law 5762 (CMVR, 2020) and Municipal Decree 14993 (PVR, 2018); F: Data available from the National Sanitation Information System (SNIS, 2021).

Table 4 -Results of the performance indicators and calculation of the General Sustainability Index.
ISSC: Indicators of sustainability of selective collection.