Filter by topic and date
IETF 118 post-meeting survey
- Jay Daley IETF Executive Director
30 Nov 2023
IETF 118 Prague was held 4-10 November 2023
The results of the IETF 118 Prague post-meeting survey are now available on a web-based interactive dashboard. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.
Analysis
We received 336 responses, all of whom participated in IETF 118, 291 onsite and 45 remote. As no the respondents did not participate in IETF 118 the specific questions for them are not shown in the dashboard. With 1806 registered participants, this gives the survey a maximum margin of error of +/- 4.82% (4.9% for onsite and 14.17% for remote).
The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.
Question changes since the last survey
For this survey we added some new questions about side meetings as this remains an ongoing area of concern. Some questions were removed (e.g. satisfaction with 'new' WGs) to keep the survey of a reasonable length. Not asking about the venue overall was an inadvertent omission.
Actions taken following the last survey
For this meeting, we made the following changes, prompted by survey feedback:
- Extended the meeting to a full day on Friday to provide for more sessions, partly to ease pressure on sessions and partly to give more options to avoid conflicts.
- Added a new Farewell Reception after the meeting closed on Friday to provide an opportunity for people to unwind and say goodbye.
- Added an alcohol-free event in the form of a movie night.
Unfortunately, this meeting did not have a meeting host and so there were no free t-shirts and no social event.
Satisfaction
Overall satisfaction is 4.42 which is a very good result. With some key exceptions, the satisfaction scores remain high, reflecting the various improvements made since we returned to onsite meetings.
The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)
| IETF 118 Prague | IETF 117 San Francisco | IETF 116 Yokohama | IETF 115 London | IETF 114 Phila. | IETF 113 Vienna | |
|---|---|---|---|---|---|---|
| Overall satisfaction | 4.42 🟢 | 4.30 🟢 | 4.30 🟢 | 4.28 🟢 | 4.19 🟢 | 4.36 🟢 |
| AGENDA | ||||||
| Overall agenda | 4.20 🟢 | 4.16 🟢 | 4.18 🟢 | 4.22 🟢 | 4.06 🟢 | 4.16 🟢 |
| Sessions for new WGs | - | 4.19 🟢 | 4.17 🟢 | 4.12 🟢 | 4.15 🟢 | 4.18 🟢 |
| Sessions for existing WGs | - | 4.22 🟢 | 4.22 🟢 | 4.22 🟢 | 4.10 🟢 | 4.24 🟢 |
| Sessions for WGs | 4.32 🟢 | - | - | - | - | - |
| BOFs | 4.12 🟢 | 3.95 🟡 | 4.11 🟢 | 4.10 🟢 | 4.09 🟢 | 4.04 🟢 |
| Sessions for existing RGs | - | 4.12 🟢 | 4.14 🟢 | 4.10 🟢 | 3.95 🟡 | 4.13 🟢 |
| Sessions for RGs | 4.09 🟢 | - | - | - | - | - |
| Plenary | 3.94 🟡 | 3.99 🟡 | 3.98 🟡 | 3.98 🟡 | 3.98 🟡 | 3.94 🟡 |
| Side meetings | 3.79 🟡 | 3.75 🟡 | 3.73 🟡 | 3.81 🟡 | 3.73 🟡 | 3.52 🟡 |
| Hackathon | 4.27 🟢 | 4.25 🟢 | 4.34 🟢 | 4.35 🟢 | 4.30 🟢 | 4.09 🟢 |
| HotRFC | 3.90 🟡 | 3.89 🟡 | 3.84 🟡 | 4.21 🟢 | 3.94 🟡 | 4.17 🟢 |
| Pecha Kucha | 4.00 🟢 | 4.15 🟢 | - | - | - | - |
| Office hours | 4.07 🟢 | 3.98 🟡 | 4.23 🟢 | 4.00 🟢 | 4.09 🟢 | 3.96 🟡 |
| Opportunities for social interaction | 4.06 🟢 | 4.11 🟢 | 3.72 🟡 | 3.98 🟡 | 3.89 🟡 | 3.51 🟡 |
| STRUCTURE | ||||||
| Overall meeting structure | 4.34 🟢 | 4.28 🟢 | 4.28 🟢 | 4.28 🟢 | 4.19 🟢 | 4.26 🟢 |
| Start time | 4.49 🟢 (9:30am) | 4.28 🟢 (9:30am) | 4.16 🟢 (9:30am) | 4.28 🟢 (9:30am) | 4.20 🟢 (10:00am) | 4.12 🟢 (10:00am) |
| Finish at 5pm Friday | 3.65 🟡 | - | - | - | - | - |
| Length of day | 4.20 🟢 | 4.30 🟢 | 4.30 🟢 | 4.32 🟢 | 4.10 🟢 | 4.20 🟢 |
| Number of days | 4.18 🟢 (5+2) | 4.27 🟢 (5+2) | 4.30 🟢 (5+2) | 4.32 🟢 (5+2) | 4.30 🟢 (5+2) | 4.23 🟢 (5+2) |
| Session lengths | 4.38 🟢 (60 / 90 / 120) | 4.41 🟢 (60 / 90 / 120) | 4.36 🟢 (60 / 90 / 120) | 4.32 🟢 (60 / 90 / 120) | 4.25 🟢 (60/120) | 4.31 🟢 (60/120) |
| Break lengths | 4.38 🟢 (30/90) | 4.32 🟢 (30/90) | 4.38 🟢 (30/90) | 4.36 🟢 (30/90) | 4.25 🟢 (30/90) | 4.16 🟢 (30/60) |
| Number of parallel tracks | 3.94 🟡 (8) | 4.08 🟢 (8) | 4.01 🟢 (8) | 3.90 🟡 (8) | 3.86 🟡 (8) | 3.92 🟡 (8) |
| PARTICIPATION MECHANISMS | ||||||
| Meetecho | 4.47 🟢 | 4.35 🟢 | 4.45 🟢 | 4.45 🟢 | 4.23 🟢 | 4.36 🟢 |
| Gather | - | 3.52 🟡 | 3.46 🔴 | 3.37 🔴 | 3.06 🔴 | 3.04 🔴 |
| Zulip | 3.94 🟡 | 3.66 🟡 | 3.77 🟡 | 3.73 🟡 | 3.56 🟡 | 2.91 ⚫️ |
| Jabber | - | - | - | - | - | 3.80 🟡 |
| Audio streams | 4.27 🟢 | 4.02 🟢 | 4.21 🟢 | 4.04 🟢 | 4.05 🟢 | 4.14 🟢 |
| YouTube streams | 4.36 🟢 | 4.32 🟢 | 4.36 🟢 | 4.25 🟢 | 4.22 🟢 | 4.25 🟢 |
| CONFLICTS | ||||||
| Conflict avoidance | 4.00 🟢 | 3.90 🟡 | 3.94 🟡 | 3.91 🟡 | 3.78 🟡 | 3.89 🟡 |
| VENUE & ACCOMM | ||||||
| Overall accommodation | 4.34 🟢 | 4.07 🟢 | 4.09 🟢 | - | - | - |
| Overall venue | - | 3.90 🟡 | - | - | - | - |
| Location | 4.52 🔵 | 3.60 🟡 | - | - | - | - |
| Venue facilities | 4.43 🟢 | 4.07 🟢 | - | - | - | - |
| Cost of rooms | 3.54 🟡 | 2.87 ⚫️ | - | - | - | - |
| Availability of rooms | 4.32 🟢 | 4.07 🟢 | - | - | - | - |
| ONSITE | ||||||
| Overall | 4.50 🔵 | 4.29 🟢 | - | - | - | - |
| Badge collection | 4.70 🔵 | 4.69 🔵 | - | - | - | - |
| WiFi | 4.17 🟢 | 3.98 🟡 | 4.06 🟢 | 4.10 🟢 | 3.82 🟡 | - |
| QR Codes | 4.18 🟢 | 4.11 🟢 | - | - | - | - |
| Break F&B | 4.05 🟢 | 4.44 🟢 | - | - | - | - |
| Breakout seating | 3.80 🟡 | 4.08 🟢 | - | - | - | - |
| Signage | 4.15 🟢 | 4.22 🟢 | - | - | - | - |
| Coffee carts | 4.64 🔵 | 4.56 🔵 | - | - | - | - |
| Childcare | 4.12 🟢 | 4.06 🟢 | - | - | - | - |
| Welcome reception | 4.08 🟢 | - | - | - | - | - |
| Farewell reception | 4.34 🟢 | - | - | - | - | - |
Areas for improvement
Cost of rooms
We have only recently started to track this and so this stage we can only note that satisfaction is on the low side, particularly as we book meetings some time in advance.
Side meetings
While there is a slight uptick in satisfaction, though not significant, this remains an issue. For this survey we asked some satisfaction questions to help understand the problem better:
| Subject | Satisfaction score |
|---|---|
| The number of side meetings | 3.71 |
| The scheduling of side meetings | 3.51 |
| The length of side meetings | 3.93 |
| The remote participation technology in side meetings | 3.41 |
| The content of side meetings | 3.95 |
These results suggest that the problem is around the organisation and support of side meetings rather than the content. There are lots of comments about side meetings that seem to indicate that the current piecemeal approach to improvement may not be enough and more comprehensive review may be needed.
Conflict avoidance
We again follow the new process to reduce conflicts as described in our blog post, and extended the meeting on Friday afternoons. This did not lead to a statistically significant increase in satisfaction with conflict avoidance. This may be something that simply cannot be improved without adversely affecting some other aspect of the meeting, such as length of the day.
Zulip
We have seen a statistically significant uptick in satisfaction for Zulip, as we hoped we would see as people became more familiar with it. However the satisfaction score is still not high enough. The comments on Zulip vary from those with views on the integration of chat into Meetecho, and those who use Zulip directly and would prefer a different product. It is really not clear if it possible to choose another product that would get a higher satisfaction score given that there are so many different products out there that people use and prefer.
Breakout seating
We thought we had this one sorted but apparently not. Back to the drawing board to understand exactly what people want, which appears to be seats within ~10 meters of the meetings rooms.
Extending to Friday afternoon
This meeting experimented with a full day on Friday and the satisfaction score for that is only acceptable not good. Given that these sorts of changes take some time to bed in, we will monitor that for the next couple of meetings. There are a number of comments that this was not advertised enough, but it is mentioned clearly in all of our emails about the meeting organisation.
Individual comments
The individual comments cover a number of key themes, some of which have already been covered above:
- The location of the venue. Almost all of the comments here we positive, both about the city and the ease of getting there. Unfortunately, we will not be returning for several years at least as the venue now charges too much for our budget.
- Cost of registration, accommodation and travel. There are multiple comments that the cost of participating onsite is too high, particularly from those in the academic community. The various recommendations include: a more general bursary program, lower registration fees, and lower cost/quality venues.
- Food and beverages provided. While there is a notable amount of positive feedback, there is also a clear theme that the provision for vegans and vegetarians was insufficient and that more care needs to be taken. There is a specific request for batch brew coffee that will be considered and a request that the baristas provide tea, which we almost fell for.
- Crowding. There are multiple concerns expressed about the density of some events, in particular the welcome reception. The background here is that this was planned to be in a larger room but this was swapped with the Hackathon room because of the very large number of people who signed up for the Hackathon. The actual number who turned up for the Hackathon was significantly lower than the number that signed up, but even then that room was crowded. The underlying issue here is how to estimate in advance the number of people coming to the large events, which is something we have to do many years in advance in order to book the space. Adding new space nearer to the time is almost impossible because it has normally been sold by the venue to other clients.
Why people participated remotely
For the second time, we asked people who participated remotely (Q5), why they did and if they would have preferred to participate onsite (Q5a). Once again, the major factor, cited by 27 of the 45, was the lack funding to travel, all of whom would have participated onsite if they could have. This means that there are several hundred who would have participated onsite if funding was available, which should be taken into consideration when considering any possibly bursary program.
And finally
Thank you everyone who responded to this survey, your feedback is much appreciated.