Bitbake: "The metadata is not deterministic and this needs to be fixed"

Question

I'm building a Bitbake recipe and getting the following error message:

ERROR: When reparsing virtual:native:/path/to/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb.do_populate_cve_db, the basehash value changed from 0b637979bcb5db4263f9ed97497a6330 to bcd28a5efe646ed4d327fefa349f889c. The metadata is not deterministic and this needs to be fixed.

This reproduces in a clean build (after bitbake -c cleanall -c cleansstate <recipe>).

What is the reason for this error? The recipe has not been modified from the upstream version.

Which poky branch do you use? Have you updated it to the current upstream state (i.e. git pull)? — g0hl1n, Feb 20 '18 at 09:00

score 10 · Answer 1 · answered Feb 21 '20 at 18:55

10

the fix I used is go to that recipe and add an empty line in the end, that will help bitbake re-recognize the recipe

answered Feb 21 '20 at 18:55

Xiao Xiu Zhe

101
1
3

2

In particular, i added a line in the image recipe i was bitbaking, that solved. – Angelo Dureghello Oct 10 '20 at 13:14
What a hacky solution... But that was the only thing that worked!!! The problem occurred during `cleansstate` in my case... – grmmgrmm May 19 '21 at 18:05
I wonder if you could also have run `touch ` to achieve the same result, without modifying the file? – Jess Bowers Jul 14 '23 at 23:39

score 8 · Answer 2 · answered May 24 '18 at 16:55

8

The following is the yocto patch that adds this diagnostic message https://patchwork.openembedded.org/patch/133517/

Here is the commit message, explaining its reasons and a possible way to get further details for the problem:

Bitbake can parse metadata in the cooker and in the worker during builds. If the metadata isn't deterministic, it can change between these two parses and this confuses things a lot. It turns out to be hard to debug these issues currently.

This patch ensures the basehashes from the original parsing are passed into the workers and that these are checked when reparsing for consistency. The user is shown an error message if inconsistencies are found.

There is debug code in siggen.py (see the "Slow but can be useful for debugging mismatched basehashes" commented code), we don't enable this by default due to performance issues. If you run into this message, enable this code and you will find "sigbasedata" files in tmp/stamps which should correspond to the hashes shown in this error message. bitbake-diffsigs on the files should show which variables are changing.

Signed-off-by: Richard Purdie

answered May 24 '18 at 16:55

Sergio

114
1
5

3

So, what's the fix? – CivFan Mar 06 '20 at 16:44
Can someone provide an example `bitbake-diffsigs` command for this case? I found the sigbasedata file, but I don't know what other file to compare it to. – CivFan Mar 06 '20 at 17:02
1

I get it that question mentions "What is the reason for this error? ". This is answered very well here. Thanks for that. However I also expected a solution/workaround so that I can proceed with build. – Sunil Shahu Feb 10 '21 at 13:39
6

This information seems correct but is not helpful. It's the typical gobbly gook from the yocto team. They seem to have invented a new language ... when one wasn't needed. What is a cooker? what does deterministic imply? What does 'confuses things alot' mean/imply? What is hard to debug? What is basehashes? What inconsistencies? What is siggen.py? – steve Mar 29 '21 at 13:40
@steve "what does deterministic imply?" . Deterministic means two runs with the same data returns the same value. As example func sum(a, b) { return a + b; } is deterministinc. func random(seed) { ... } is not deterministic as to call the function with the same seed parameter - ence the same input data - with give different values on each run. – iagorubio Nov 08 '22 at 08:32

score 8 · Answer 3 · answered Nov 19 '20 at 09:57

8

I stumbled onto a similar error. For me it happened in the do_install routine. My aim was to store a copy of a file with a ${DATETIME} attached to its name referring to the build time in a specific place.

Apparently the recipes get parsed multiple times during a build and since the time has changed by the second time it got parsed a different ${DATETIME} value was inserted and thus the metadata was recognized as changed.

answered Nov 19 '20 at 09:57

Benedikt

81
1
4

1

@Timus for someone familiar with Yocto and Bitbake this _is_ an answer, or at least a very strong hint where to look for one, as what is describes is one of the possible causes for the issue in the question. – jaskij Jan 19 '21 at 10:33
@Timus admittedly, this answer is unusual, in that recipes normally don't include DATETIME, but it's one of the possible causes nonetheless. – jaskij Jan 19 '21 at 11:47
@Timus in case you are curious, Yocto has very strong focus on reproducible builds, to the extent that it even builds the toolchain from sources. Users should use commit hashes, source archive hashes and the like to identify which source it was built from, as in theory source distribution should enable a reproduction. It's a pretty specialized piece of coding. – jaskij Jan 19 '21 at 11:51
2

@Benedikt, so what is the proper way to do time-stamping in yocto ? – codeist Aug 31 '22 at 07:56

score 6 · Answer 4 · answered Jul 28 '21 at 08:42

6

Simplest solution: touch <recipename>. Then run cleansstate on your recipe. Then go on as usual from here.

answered Jul 28 '21 at 08:42

grmmgrmm

994
10
29

score 4 · Answer 5 · answered Oct 29 '18 at 08:47

4

It helps for me to remove a shared cache for a recipe:

bitbake recipename -c cleansstate

And then, everything works like a charm.

answered Oct 29 '18 at 08:47

AcRap

49
2

2

Not a fix for me. – CivFan Mar 06 '20 at 16:45

codeist · Accepted Answer · 2022-09-01T05:45:58.203

This happens because tasks are evaluated twice, the first time by the cooker, and the second time by bitbake worker. The task hash is calculated twice and if it will not match, meta is considered unstable. Base hash is calculated from variables that are used in the task script. So if you for example use time-related variables like DATETIME, you will get this hash mismatch error. To avoid this error you need to exclude variables from the hash calculation, with VarFlags.

do_something[vardepsexclude]="DATETIME"

here you can find more elaborate explanation. https://www.kc8apf.net/2017/01/tired-of-taskhash-mismatch/

yocto bitbake hashes

score 1 · Answer 7 · answered Jun 09 '21 at 13:37

1

For what it's worth, doing this cleared it up for me:

devtool modify <recipe>
devtool reset <recipe>

answered Jun 09 '21 at 13:37

Bill Tonnies

11
1

score 1 · Answer 8 · edited Aug 01 '21 at 20:31

1

this also happens if you modify the recipe while you're building...

this means that the error you stated goes off if you edit a recipe file while a build is in progress. If this is your case just finish your edit and re-launch a build, it should work. Happened to me and solved this way.

edited Aug 01 '21 at 20:31

Jean-François Fabre

137,073
23
153
219

answered Jul 06 '21 at 14:50

c_Ake

11
3

score 0 · Answer 9 · answered Mar 30 '18 at 14:51

0

This happens when a recipe contains changing data and you try to rebuild it, one way to avoid this is to delete the tmp/ directory [not a good solution] but it is the least possible solution.

answered Mar 30 '18 at 14:51

Sajjad Ahmad

421
6
13

9

What means "the recipe contains changing data"? Which "data"? – Konstantin Shemyak Mar 30 '18 at 18:49

Bitbake: "The metadata is not deterministic and this needs to be fixed"

9 Answers9