user/erlang: FTTFS: peer:start_it failed on line 929 / Reason: timeout

marked this issue as related to #1226 (closed)

Fixed by (for example):

--- etc/hosts.orig      2024-07-25 18:48:02.504527773 -0500
+++ etc/hosts   2024-07-25 18:37:03.382681677 -0500
@@ -1,2 +1,2 @@
-127.0.0.1      localhost
-::1            localhost
+127.0.0.1      localhost ppc64
+::1            localhost ppc64

This appears to be mitigated when: the output of uname -n is present in /etc/hosts, regardless of protocol (IPv4 or IPv6; only one is needed).

The contents of /etc/hostname do not matter, period.

That was introduced by one or both of:

As for why this did not crop up in Erlang before we updated? TBD.

mentioned in issue #1226 (closed)

This doesn't make a whole lot of sense, or rather, I think this is a workaround and not the root cause, because:

awilcox on lab-arm64-lin-1 /opt/testroot % uname -n
lab-arm64-lin-1.tpcl.atwilcox.tech
awilcox on lab-arm64-lin-1 /opt/testroot % ping lab-arm64-lin-1.tpcl.atwilcox.tech
ping: lab-arm64-lin-1.tpcl.atwilcox.tech: Name does not resolve

For kicks, I did the following on the host machine, to check if a . was expected to be present (as opposed to just ppc64):

hostname -b ppc64.test

The tests FAILED.

Next, I did this to match yours, to rule out any weirdness:

hostname -b lab-arm64-lin-1.tpcl.atwilcox.tech

The tests FAILED.

Then, inside the autobuilder rootfs, I appended lab-arm64-lin-1 (not the full FQDN) to the first (IPv4) line of /etc/hosts.

The tests PASSED.

Then, I edited that file to use only the hostname -s output, such that /etc/hosts inside the rootfs still contained the FQDN.

The tests FAILED.

So I amend my previous statement that uname -n is needed; it is actually hostname -s that appears to need to be resolvable.

I also tried adding this to /etc/hosts in the rootfs:

+ 127.0.1.1 ppc64

The tests PASSED.

I also tried to collect more information on what does or does not resolve:

(inside the autobuilder debug environment, with non-root privileges)

where (generated by autobuilder on first run):

$ cat /etc/hosts
127.0.0.1 localhost
::1 localhost

and (provided by DHCP during initial setup of the host):

$ cat /etc/resolv.conf
domain vpn.adelielinux.org
search vpn.adelielinux.org
nameserver 192.168.10.1

I am using a minimal ping utility for a quick test:

$ curl -so ping https://landley.net/toybox/bin/toybox-powerpc64
$ chmod +x ping

and conduct these tests:

$ for k in localhost www.adelielinux.org $(hostname -s); do ./ping -4c1 ${k}; done
Ping localhost (127.0.0.1): 56(84) bytes.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=0 time=0 ms

--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms
Ping www.adelielinux.org (216.82.192.11): 56(84) bytes.
64 bytes from 216.82.192.11: icmp_seq=1 ttl=0 time=0 ms

--- 216.82.192.11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms
Ping ppc64 (216.82.192.11): 56(84) bytes.
64 bytes from 216.82.192.11: icmp_seq=1 ttl=0 time=0 ms

--- 216.82.192.11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms

It seems to resolve ppc64 to 216.82.192.11, which is the WAN address of our web server, due to the domain and search lines above. The hostname vpn.adelielinux.org does not exist, but the CNAME *.adelielinux.org goes to sentry.adelielinux.org which is our web server.

I removed those two lines from /etc/resolv.conf:

$ hostname -s
ppc64
$ ./ping $(hostname -s)
ping: ppc64: Name has no usable address

and now tests PASS.

So the root cause appears to be, hostname -s must either NOT resolve, or resolve to the local machine, which is to say that the search keyword needs to not point to something that will return a remote IP address when hostname -s is prepended to it:

Tests PASS with this:

nameserver 192.168.10.1

Therefore I propose that autobuilder the script do some sort of filtering on /etc/resolv.conf when it copies this file from the host into the rootfs, as in, strip anything that isn't nameserver.

TODO:

Determine why this changed between previous Erlang releases (should be easy enough to bisect);
Determine whether this is as-designed or if this needs to be reported upstream.
Determine why Erlang test failures DO NOT cause the build to ultimately fail.

mentioned in issue adelie-infra/autobuilder#49 (closed)

marked this issue as related to adelie-infra/autobuilder#49 (closed)

24.3.3 (our previous version) waits approximately 168 seconds before failing when search is set, while 27.0.1 waits approximately 5-10; this MAY be due to this change, which was first released in 26.0.

It doesn't materially affect the test suite. Versions between 24.3.3 and that commit do fail faster, and I can't be bothered to bisect it right now. 24.0.1 (earliest version with configure in the repository) is also affected. Earlier versions may require autotools.

Since the search issue affects 24.0.1 as well, and likely earlier versions, it is safe to say that the issue was masked by (3) in the previous comment.

Removing search in the above examples causes the test suites to run as expected.

Recommend splitting into separate issues.

If you want to go back earlier, release tarballs of 21.2 appear to have not needed autotools. We added Erlang at version 21.2 in 5ba732fc. It was promoted from experimental/ to user/ in 93f4cd8a. It built, and passed tests, on x86_64 and ppc64, at that time. The tests had failed in the past, but were resolved by other package updates. It looks like it was possibly a musl update, actually…

marked this issue as related to #1229

TODO:

Determine whether this is as-designed or if this needs to be reported upstream.

user/erlang: FTTFS: peer:start_it failed on line 929 / Reason: timeout

Designs

Child items ...

Activity