Empty group and unexpected results from real pol.is data example

[nicobao]

Hi Erik,

I tried running polilite with a real pol.is conversation:

• https://pol.is/664akjpxey
• https://pol.is/report/r2dtdbwbsrzu8bj8wmpmc

I converted the two .csv to your expected .yaml format report_from_polis.yaml.

Repro available here: https://github.com/nicobao/polislite-repro

As you can see in result.log, it's strange:

Consensus Statements:
- Our nearby city comes to our brigade with internal (but open souce) projects that need a new stakeholder to reach their potential. (strong disagreement)
- My brigade holds my local government accountable (strong disagreement)
- My brigade gives a voice to people impacted by government services (strong disagreement)
- My brigade offers workforce development for my local community (strong disagreement)
- My brigade advises local government and non-profit partners (strong disagreement)
- My brigade trains locaal government and non-profit partners (strong disagreement)
- My brigade provides volunteers that resource projects from other local initiatives (strong disagreement)
- My brigade provides volunteers that resource projects from other brigades (strong disagreement)
- This is a substantial focus of our work. (strong disagreement)
- I want my brigade to be seen as a go-to resource for digital expertise among local government staff. (strong disagreement)
- The “principles & practices of the digital age” embed inequity and bias into our technology. I want brigades to work toward changing this. (strong disagreement)
- Brigades' focus should be on improving tech. delivery in government, and are not well positioned to try to be everything to everyone. (strong disagreement)
- A technologist's role in community problems is to democratize access to technology (strong disagreement)
- A year from now, our brigade is likely to have more active local members. (strong disagreement)
- CfA should partner with existing open source projects for good, like Mozilla Fdn, Wikimedia, etc (strong disagreement)
- Our brigade serves as a matchmaker, connecting stakeholders from different sectors and fields who may have shared interests. (strong disagreement)
- My brigade spends too much time reinventing the wheel instead of iterating and innovating. (strong disagreement)
- Members of my brigade would appreciate being matched to projects that fit their interests and skills, even if they weren't local (strong disagreement)
- Maintaining high-quality software projects long-term without the support of paid staff is not realistic for my brigade. (strong disagreement)
- We should be spending more time working with and educating elected officials. (strong disagreement)
- My Brigade wants to have a sustainable funding model. (strong disagreement)
- My brigade wants local private tech companies to sponsor us. (strong disagreement)
- My brigade has the capacity to pursue multiple projects. (strong disagreement)
- Not formally, but how we do everything: learning through doing. (strong disagreement)
- My brigade has more positive impact by helping participants better understand government and civics, than by building tech tools. (strong disagreement)
- I want my brigade to deliver quickly enough on important projects so our community partners succeed, appreciate us, and brag on us. (strong disagreement)
- Our community needs a brigade because our city IT department uses every excuse to gatekeep and build power for themselves over the people. (strong disagreement)
- I believe, if we work together, we can find thoughtful ways to advance diversity/equity/inclusion without sacrificing transparency. (strong disagreement)
- My brigade is getting better at working with people-powered community partners. (strong disagreement)
- My brigade has said they want to share their work/learnings, since every insight & every code snippet has the power to make us all better. (strong disagreement)
- My brigade, OpenOakland, has become a slave to require projects to be politically correct.  Not just do good governement. (strong disagreement)
- My brigade has become preoccupied with requiring projects to be politically correct. Not just do good government. (strong disagreement)
- I want my brigade to leverage data to build understanding and trust between our community and our government. (strong disagreement)
- My brigade struggles to complete projects. (strong disagreement)
- I want my brigade to practice age equity in its recruitment to achieve diversity of experience in its project teams. (strong disagreement)
- My brigade members have said they just want to work on a specific project. (strong disagreement)
- My brigade struggles to maintain more than 6 active members at a time. (strong disagreement)
- My brigade leadership is overwhelmed by the amount of work to be done. (strong disagreement)
- My brigade leadership does not enjoy doing organizing work. (strong disagreement)
- My brigade leadership does not enjoy doing fundraising work. (strong disagreement)
- I want my brigade to be better at storytelling. (strong disagreement)
- My brigade needs help recruiting members. (strong disagreement)
- My brigade is majority technologists with little to no civics understanding. (strong disagreement)
- My brigade doesn’t value the time or effort of its members and will discard work that is done by them. (strong disagreement)
- Our local government is focused on internnal performance KPIs and not projects in the public interest. (strong disagreement)
- Our brigade has been most successful when we've partnered with local nonprofits and social workers. (strong disagreement)
- The nonprofit community has a much better understanding of the needs of individuals and families seeking help from government resources. (strong disagreement)
- Code schools / bootcamps are a great recruiting source for brigade volunteers. We don't teach code, but we introduce students to civic tech. (strong disagreement)
- I want to be part of a National Brigade that fixes systemic federal government problems. (Even if fed gov is not asking for help). (strong disagreement)
- I believe brigades can overcome social inequality and economic inequality utilizing various resources at their disposal. (strong disagreement)
- I want my brigade to organize with others in our community so that technology (Facial recognition) isn't used to discriminate in our city. (strong disagreement)
- My brigade cannot align our work with CfA's projects because we don't know what CfA is doing or how we can collaborate. (strong disagreement)
- My brigade is struggling to deliver value to partners *and* to enable volunteers to engage productively. (strong disagreement)
- More than anything else, our volunteers are motivated by personal, tangible contributions to meaningful work. (strong disagreement)
- More than anything else, our volunteers are motivated by being part of a like-minded community. (strong disagreement)
- More than anything else, our volunteers are motivated by highly-impactful projects. (strong disagreement)
- Our volunteers are DE-motivated by periods of inaction more than anything else. (strong disagreement)
- Money is one of the top issues preventing progress in my brigade. (strong disagreement)

Divisive Statements:
- My brigade has fewer than 10 active members
- My brigade has more than 50 active members
- I attended my first brigade meeting over 2 years ago.
- My brigade changes the conversation locally through long-term government partnerships
- My brigade creates new civic technologists.
- My brigade trains future public servants with digital delivery skills.
- My brigade solves local problems with the tools of technology, locally.
- My brigade funnels new and emerging technologists into “good tech”
- My brigade provides digital services and capacity to local government starved for these resources.
- My brigade provides digital services and capacity to local nonprofits starved for these resources.
- My brigade scales Code for America products and principles.
- My brigade advocates for good technical policy and against bad policy.
- My brigade volunteers serve as the thought leaders and practitioners of the civic technology ecosystem.
- My brigade trains others (technologists, gov staff, nonprofits, activists) on how to apply the principles and practices of the digital age.
- My brigade wants to lift up the science and technology of indigenous people
- I want my brigade to push back on local government when we see systemic inefficiencies and inequities in service delivery.
- I want my brigade to serve as a partner and champion of government partners.
- I want my brigade to serve as a partner & champion for traditionally marginalized groups who are otherwise underserved by local government.
- I want my brigade to be seen as a go-to resource for digital expertise among local community groups.
- I want my brigade to develop tech solutions that are scalable across our local government partners.
- I want my brigade to develop technology that expands the impact of local social justice groups.
- I want my brigade to augment or expand our local government’s digital capacity.
- I want my brigade to use tech to elevate & amplify the intrinsic knowledge of our local marginalized & underrepresented communities.
- I want my brigade to deliver end-to-end technology solutions to local government, similar to how a vendor might.
- I want my brigade to partner with and learn from local government when we see systemic inefficiencies and inequities in service delivery
- I want my brigade to raise the visibility of issues of equitable service delivery in our community.
- I believe brigades are most effective when working in partnership with local, community based organizations.
- I believe brigades must engage more actively in advocacy to have long-term change on the problems we seek to address
- My brigade does projects as experiments that push government to view its own mandate in those areas differently.
- My brigade doesn't do enough to center the voices of those our technology most impacts.
- Technologists aren't always the best people to solve community problems.
- My brigade prioritizes amiable government partnerships over community needs.
- The "principles and practices of the digital age" have different meaning at local vs national government, nonprofits, and tech companies
- Brigades are not a place for folks to learn to code, but for those with skills to give back.
- My brigade receives enough support from Code for America.
- My brigade needs funding or assistance with fundraising to reach our full potential
- I'd like my brigade to find a better way to use the adjacent policy expertise we often attract but can't retain.
- My Brigade wants to run regular events in more areas of our city
- My brigade wants volunteers from local private technology companies.
- Principles, yes; products, not generally.
- Occasionally; also we work with several different local cities due to the nature of the geography of local munis.

Group Positions:
Group 1 characteristics:
Group 2 characteristics:
- strongly disagrees with: My brigade has fewer than 10 active members
- strongly agrees with: My brigade creates new civic technologists.
- strongly agrees with: My brigade solves local problems with the tools of technology, locally.
- strongly agrees with: My brigade provides digital services and capacity to local nonprofits starved for these resources.
- strongly agrees with: My brigade scales Code for America products and principles.
- strongly agrees with: I want my brigade to push back on local government when we see systemic inefficiencies and inequities in service delivery.
- strongly agrees with: I want my brigade to serve as a partner and champion of government partners.
- strongly agrees with: I want my brigade to serve as a partner & champion for traditionally marginalized groups who are otherwise underserved by local government.
- strongly agrees with: I want my brigade to be seen as a go-to resource for digital expertise among local community groups.
- strongly agrees with: I want my brigade to develop tech solutions that are scalable across our local government partners.
- strongly agrees with: I want my brigade to develop technology that expands the impact of local social justice groups.
- strongly agrees with: I want my brigade to augment or expand our local government’s digital capacity.
- strongly agrees with: I want my brigade to use tech to elevate & amplify the intrinsic knowledge of our local marginalized & underrepresented communities.
- strongly agrees with: I want my brigade to partner with and learn from local government when we see systemic inefficiencies and inequities in service delivery
- strongly agrees with: I want my brigade to raise the visibility of issues of equitable service delivery in our community.
- strongly agrees with: I believe brigades are most effective when working in partnership with local, community based organizations.
- strongly agrees with: CfA should partner with existing open source projects for good, like Mozilla Fdn, Wikimedia, etc
Group 3 characteristics:
- strongly disagrees with: My brigade has more than 50 active members
- strongly agrees with: I attended my first brigade meeting over 2 years ago.
- strongly agrees with: My brigade creates new civic technologists.
- strongly disagrees with: My brigade provides digital services and capacity to local government starved for these resources.
- strongly agrees with: I want my brigade to push back on local government when we see systemic inefficiencies and inequities in service delivery.
- strongly agrees with: I want my brigade to serve as a partner & champion for traditionally marginalized groups who are otherwise underserved by local government.
- strongly agrees with: I want my brigade to be seen as a go-to resource for digital expertise among local community groups.
- strongly agrees with: I want my brigade to develop tech solutions that are scalable across our local government partners.
- strongly agrees with: I want my brigade to augment or expand our local government’s digital capacity.
- strongly agrees with: I want my brigade to partner with and learn from local government when we see systemic inefficiencies and inequities in service delivery
- strongly agrees with: I want my brigade to raise the visibility of issues of equitable service delivery in our community.
- strongly agrees with: I believe brigades are most effective when working in partnership with local, community based organizations.
- strongly agrees with: My brigade does projects as experiments that push government to view its own mandate in those areas differently.
- strongly agrees with: My brigade doesn't do enough to center the voices of those our technology most impacts.
- strongly disagrees with: Technologists aren't always the best people to solve community problems.
- strongly agrees with: My brigade prioritizes amiable government partnerships over community needs.
- strongly agrees with: The "principles and practices of the digital age" have different meaning at local vs national government, nonprofits, and tech companies
- strongly agrees with: CfA should partner with existing open source projects for good, like Mozilla Fdn, Wikimedia, etc
- strongly agrees with: Members of my brigade would appreciate being matched to projects that fit their interests and skills, even if they weren't local
- strongly agrees with: We should be spending more time working with and educating elected officials.
- strongly agrees with: My Brigade wants to run regular events in more areas of our city
- strongly agrees with: My Brigade wants to have a sustainable funding model.
- strongly agrees with: My brigade wants volunteers from local private technology companies.
- strongly agrees with: My brigade has the capacity to pursue multiple projects.
- strongly agrees with: Not formally, but how we do everything: learning through doing.
- strongly agrees with: Our community needs a brigade because our city IT department uses every excuse to gatekeep and build power for themselves over the people.
- strongly agrees with: My brigade is getting better at working with people-powered community partners.
- strongly agrees with: My brigade has said they want to share their work/learnings, since every insight & every code snippet has the power to make us all better.
- strongly disagrees with: My brigade has become preoccupied with requiring projects to be politically correct. Not just do good government.
- strongly agrees with: My brigade struggles to complete projects.
- strongly agrees with: I want my brigade to practice age equity in its recruitment to achieve diversity of experience in its project teams.

Any idea what went south?

[zkYuting]

Not sure why group 1 is empty. The "consensus statements" and "divisive statements" seem to contradict the group characteristics. For example, statement 26 "I want my brigade to serve as a partner & champion for traditionally marginalized groups who are otherwise underserved by local government." is strongly agreed by both Group 2 & 3 but shows up in divisive statements.

[eterps]

Hi @nicobao

Great idea to verify it against actual Pol.is data. I didn't know an easy way to do that, but this is excellent.

Any idea what went south?

The vote encoding might need adjustment, especially "pass" values, which could skew the results.
Patcon actually proposed that change here: #3

So my code is probably not accounting for the large number of "pass" votes
And isn't distinguishing between genuine disagreement and lack of votes

Now that you have actual pol.is data for comparison you might also want to check it against the code that Maanas created which is arguably closer to the algorithm that pol.is itself uses: https://github.com/MaanasArora/polis-ctto/blob/master/polis.server/polis/core/routines.py (its probably not very hard to substitute the algorithm for testing it against your sample data).

[nicobao]

I would love to experiment with poli-ctto, but there is no open-source license :/
Thanks for your reply on your code! Very interesting!

[eterps]

I would love to experiment with poli-ctto, but there is no open-source license :/

That shouldn't prevent you from testing it in the first place, no need for an open-source license if it doesn't produce the results you want 😉

[nicobao]

Btw I tried adding Patcon suggested fix, but it doesn't change the result!
#3 (comment)

(I had actually already merged the add_test branch to main, that's what I tested the data against)

[eterps]

I hadn't tried Patcon's suggested fixes because I started my new job recently. I would have to check again during the weekend how to handle 'pass' votes.

[nicobao]

No problem. I don't have any expectations whatsoever :)
(And if I really want it's my responsibility to fix it myself, however I am looking to compare solutions first, and we have a tight deadline ☠️)

I hope you love your new job!!

polis-ctto has a license now. That was fast! I'm trying this now!

[nicobao]

After careful testing of both this repo and polis-ctto, I came to the conclusion that the core Pol.is algorithm has so many hidden details built over years of testing that makes the clustering particularly good.

For example, they clean the data to prepare for PCA:
https://github.com/zkorum/polis-wl/blob/c79d47bf93d9582d90b767219f9d2080ac4f8641/math/src/polismath/math/pca.clj#L108-L124

It seems that all these details are critical to end-up with relevant clusters at scale

[nicobao]

Here is my attempt to combine your repo with polis-ctto (K-MEANS):

I tested multiple combinations. polis-ctto get_kmeans function somehow failed during the matrix division, saying it was dividing by 0... So I changed the PCA alg though.

https://github.com/zkorum/polislite/blob/with-kmeans/polis_core.py#L35-L36
https://github.com/zkorum/polislite/blob/with-kmeans/polis_core.py#L116-L132