Question about metric implementation

[marcosmacedo]

Hi,

Thank you for sharing your CodeBLEU package. Could you please explain why is there an or 1 expression in the condition below in your implementation?

+ theta * (dataflow_match_score or 1)

codebleu/codebleu/codebleu.py

Line 87 in 4b8d576

+ theta * (dataflow_match_score or 1)

This is different from the XLCoST implementation.

https://github.com/reddy-lab-code-research/XLCoST/blob/ad46a7df51ea9e88f37a2f7e6edc5cbe4d13b2f2/code/translation/evaluator/CodeBLEU/calc_code_bleu.py#L76

During my testing I get a CodeBLEU score of 0.25 even if all the weights of the metric are zero. Is that the intended behavior?

Thank you

[k4black]

@marcosmacedo Hey, thank you for spotting this;
TBH i do not remember =)
If im not mistaken it relay on examples from the original articles, but let me check

On it

[F2011]

@k4black, are you still working on this? Would be interesting to hear why you implemented it that way. It seems like you did it for a specific reason, but I could not find any hint on it - neither in the CodeBLEU paper, nor in Microsoft's implementation