Getting started

  • /submission: Submit your model output
  • /study: Study screen for human evaluation
  • /user: User participation
  • /result: Leaderboard

Summary

System naming convention

  • natural mocap data (ground truth): NA
  • submitted systems: SA, SB, ..., SZ
  • baseline systems: BA, BB, BC, ...

We provide a input and output to the participants, and you generate gestures npy for the input.

Team ATeam BTeam Z
Input 1
Input 1
Input 2
Input 2
Input 3
Input 3
Input 4
Input 4Model SAModel SBModel SZMotion SA_1Motion SA_2Motion SA_3Motion SA_4Motion SB_1Motion SB_2Motion SB_3Motion SB_4Motion SZ_1Motion SZ_2Motion SZ_3Motion SZ_4Ground TruthBaseline BA
$$\cdo...
Video SA_1Video SA_2Video SA_3Video SA_4Video SB_1Video SB_2Video SB_3Video SB_4Video SZ_1Video SZ_2Video SZ_3Video SZ_4Video BA_1Video BA_2Video BA_3Video BA_4Video NA_1Video NA_2Video NA_3Video NA_4Motion BA_1Motion BA_2Motion BA_3Motion BA_4Motion NA_1Motion NA_2Motion NA_3Motion NA_4

Then, we recruit Prolific to start user studies. Using pairwise comparison, we evaluate the generated gestures with human evaluation

Video SA_1Video SA_2Video SA_3Video SA_4Video SB_1Video SB_2Video SB_3Video SB_4Video SZ_1Video SZ_2Video SZ_3Video SZ_4Video BA_1Video BA_2Video BA_3Video BA_4Video NA_1Video NA_2Video NA_3Video NA_4Participant 1Participant 2Participant 3Participant 4ScreenSA_3 - SB_3ScreenNA_2 - SZ_2ScreenSA_4 - BA_4
emotion mismatching studies
emotion mismatching studiesScreenSZ_3 - BA_3ScreenSA_4 - SZ_4ScreenSA_2 - NA_2
speech appropriateness studies
speech appropriateness studiesScreenSA_1 - BA_1ScreenSA_2 - SB_2ScreenSC_3 - NA_3
pairwise emotion studies
pairwise emotion studiesScreenBA_4 - SB_4ScreenBA_2 - SZ_2ScreenNA_1 - BA_1
pairwise human-likeness studies
pairwise human-likeness studiesResult Study 1Result Study 2Result Study 3Result Study n
Score SAScore SBScore SZ

Evaluation Process

Download submission_inputs.csv file

input_code, video_file
234083450345, 234083450345.npy
346643424234, 346643424234.npy
443646423423, 443646423423.npy

Run your model inference

Run your model inference to get all video file output.

234083450345.npy
346643424234.npy
443646423423.npy

Submission your model .npy output

Go to /submission and upload all video of your model

Convert & Split to Video

We will convert all .npy file to .npy video and split to mutiple segment video.

Generate all study pairwise compare screen

Sample study screen

study_screen
  • You can visit /study to get information about each study screen.
  • You can visit /user to follow all user participation our study.

Recruit partipation on Prolific to study

Each Prolific partipation will be study for

prolific

When partipation study, we record all action and final result

Selection Result

selection_result

Actions Record

actions_record

Evalution final result

Sample evalution result

eval_result