Evaluating Pairlab.io as a tool for community feedback

I've created a demo survey here: https://app.pairlab.io/s/80849

And tweeted about it: https://twitter.com/TimLehnen/status/1682141420923265026

I took the survey. It was far more pleasant than most surveys I have taken!

As a user of the survey I liked:

1. The Tour was very helpful and clear.
2. Comparing two options at a time. That is much easier to think about than trying to rank more than 2 options.
3. Able to stop whenever you wanted to. Can one resume later?

But did not like:

1. I did not know how many 'topics' were being evaluated
2. The links in the blue boxes were not clickable nor could I copy them

It does 'feel' like a useful tool and I am looking forward to seeing what the results look like. I do wonder, though, how it can be abused.

I like the possibility to revisit the gathered data down the road. But we need to acknowledge that it will be their data, not the Drupal community's. Not sure if that is a big enough concern for everyone, but I thought to highlight it.

I read the privacy policy. I am no lawyer or expert in the matter, I am just another privacy aware community member.

In my opinion, the current policy is not great, but it isn't abusive either. Concerning quotes include:

We may disclose your information to third parties when: [...] It’s required by third parties to help us deliver services to you and support our operations

In my experience, this typically translates to, we'll disclose your data whenever.

The biggest issue IMHO is this very common, and very abusive, clause:

We may update or change this policy at any time. Your continued use of this site after any change indicates your acceptance of such changes.

By participating we accept any possible changes, abusive or not.

I like the tool itself, and I fully acknowledge that no tool will be perfect. The perfect homegrown tool will also come with a maintenance burden "in perpetuity".

My vote is a hesitant +1. I don't have a better alternative. My hesitation mainly comes from the abusive clause highlighted earlier.

@quietone - I'll definitely pass that feedback on to the developer.

@e0ipso - because the tool is built by a Drupal community member, I'm betting that the current terms of service are mostly a boilerplate example.

I suspect they might be willing to update those terms, or sign a separate data agreement with the DA, or perhaps even let us host our own instance.

Thanks for taking the time to give feedback, it helps a lot.

I also struggled a bit with non-clickable links in buttons. I would love if the help text for each option was available so I could learn more if needed.

Thanks for all the comments (pairLab is my project). To address them:

• I didn't fully understand hestenet's use case about linking ideas out to explanations. I understand now and am thinking about a fix.

• @quietone, yeah, that makes sense -- I'll quantify the number of ideas in the tour.

• @e0ipso -- the privacy policy is boilerplate. I don't have a plan to disclose any data. I'm happy to change it and I'm open to a data agreement, as hestenet suggests

I built pairLab in Drupal. If you're interested in knowing more about what I did, take a look at the DrupalCon Global 2020 session I presented: https://www.youtube.com/watch?v=YaJYYUW_8mc

@quietone -- I forgot to address the potential for abuse you raised. Posting a survey link publicly can invite abuse. We could try to mitigate that by sending the invite link by email, though the email can be shared.

On the data side, I have some anti-spam measures in place. I also have a data scrubbing feature, which looks for a few things that could indicate abuse:

• pairLab flags a survey participant who skipped too many challenges and who gave too few responses before they stopped. These flags can indicate that the participant might be trying to game ideas they either favor or disfavor, or that they didn't understand ideas they saw.

• pairLab scores a participant's aggregate responses by comparing them against a set of purely random responses. When the two are similar, it suggests the participant randomly clicked around or might not have understood the presented ideas.

In both cases, a participant's votes can be excluded from the results.

Also, I assign a uuid to each participant at the beginning of the survey. While that ensures participants and their votes are anonymous, if someone takes the survey again they are assigned a new uuid and are counted as a new participant. I'm not sure if that's an abuse, but it's a thing.